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Abstract. Originally motivated by algebraic invariant theory, we present an 
algorithm to enumerate integer vectors modulo the action of a permutation 
group. This problem generalizes the generation of unlabeled graph up to an 
isomorphism. In this paper, we present the full development of a generation en- 
gine by describing the related theory, establishing a mathematical and practical 
complexity, and exposing some benchmarks. We next show two applications 
to effective invariant theory and effective Galois theory. 

Initialement motive par la theorie algebrique des invariants, nous presen- 
tons une strategie algorithmique pour enumerer les vecteurs d'entiers modulo 
Taction d'un groupe de permutations. Ce probleme generalise le probleme 
d'enumeration des graphes non etiquetes. Dans cet article, nous develop- 
pons un moteur complet d'enumeration en expliquant la theorie sous-jacente, 
nous etablissons des bornes de complexite pratiques et theoriques et exposons 
quelques bancs d'essais. Nous detaillons ensuite deux applications theoriques 
en theorie effective des invariants et en theorie de Galois effective. 

Generation up to an Isomorphism, Enumerative Combinatorics, Computational 
Invariant Theory, Effective Galois Theory 

1. Introduction 

Let G be a group of permutations, that is, a subgroup of some symmetric group 
& n . Several problems in effective Galois theory (see |Gir87[lAbd00] ). computational 
commutative algebra (see |FR091 IBT111 iBorllj ) and generation of unlabeled with 
repetitions species of structures rely on the following computational building block. 

Let N be the set of non-negative integers. An integer vector of length n is an 
element of N n . The symmetric group 6„ acts on positions on integer vectors in 
N": for a a permutation and (yi, . . . ,v n ) an integer vector, 

a.(vi,...,v n ) := (v a -i (i),..., Uo-i(n))- 

This action coincides with the usual action of &„ on monomials in the multivariate 
polynomial ring K[x] with K a field and x := xi, . . . , x n indeterminates. 

Problem 1.1. Let G C & n be a permutation group. Enumerate the integer vectors 
of length n modulo the action of G. 

Note that there are infinitely many such vectors; in practice one usually wants 
to enumerate the vectors with a given sum or content. 

For example, the Problem 11.11 contains the listing non- negative integer matrices 
with fixed sum up to the permutations of rows or columns appearing in the theory 
of multisymmetric functions [Gcs87, Mac04 and in the more recent investigations 
of multidiagonal coinvariant Bcr09, BBT11 . 
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Define the following equivalence relation over elements of N": two vectors u := 
(ai, . . . , a n ) and v := (61, ... , b n ) are equivalent if there exists a permutation a 6 G 
such that 

cru= (a CT -i(i), . . . ,a CT -i (n) ) = (&!,..., 6 n ) = v. 

Problem 11.11 consists in enumerating all N n /G equivalence classes. 

This problem is not well solved in the literature. Some applications present a 
greedy strategy searching and deleting all pairs of vectors such that the second part 
can be obtained from the first part. The most famous sub- problem is the unlabeled 
graph generation which consists in enumerate tuples over and 1 of length (™) 
enumerated up to the action of the symmetric groups acting on pair on nodes. This 
example has a very efficient implementation in Nauty which is able to enumerate 
all graphs over a small number of nodes. 

The algorithms presented in this paper have been implemented, optimized, and 
intensively tested in Sage |S + 09| . most features are integrated in Sage since release 
4.7 (2011-05-26, ticket #6812, 1303 lines of code including documentation). 

2. Orderly generation and tree structure over integer vectors 

The orderly strategy consists in setting a total order on objects before quotienting 
by the equivalence relation. This allows us to define a single representative by orbit. 
Using the lexicographic order on integer vectors, we will call a vector v canonical 
under the action of G or just canonical if v is maximum in its orbit under G for 
the lexicographic order: 

v is canonical 4$ v = max{cr ■ v | a £ G}. 

lex 

Now, the goal being to avoid to test systematically if vectors are canonical, we 
decided to use a tree structure on the objects in which we will get properties relaying 
the canonical vectors. Any result relating fathers, sons and the property of being 
canonical in the tree may allowed us to skip some canonical test. 

2.1. Tree Structure over integer vectors. Let r be the vector r := (0, . . . ,0) 

called root, we build a tree with the following function father. 

Definition 2.1. Let a = (01,02, ■ ■ ■ ,a n ) be a tuple of integers of length n which 
is not the root. Let 1 ^ i ^ n be the position of the last non-zero entry of el. We 
define the father of a 

father(a 1 ,a 2 , . . . , a,-, 0, 0, . . . , 0) := (a 1 ,a 2 , . . . , a,- - 1,0, 0, . . . , 0) 

For any integer vector v = (oi, . . . , a n ), we can go back to the generation root 
(0, . . . , 0) by sum(v) :— ai + ■ ■ ■ + a n steps. The corresponding application giving 
the children of an integer vector is thus: 

Definition 2.2. Let a = (oi, 02, ■ ■ • , a, n ) be a tuple of integers of length n. Let 
1 i n be the position of the last non-zero entry of a (i = 1 if all entries are 
null). The set of children o/a is obtained as: 



children: (ai, 0,2, ■ ■ ■ , di, 0, 0, . . . , 0) 



(ai,a 2 , . . . ,a* + 1,0, 0, ... ,0) 
(ai, a 2 , . . . ,a t , 1, 0, . . . , 0) 
(01,02, . . . ,aj,0, 1, . . .,0) 

(01,02, • ■ • ,Oj, 0, 0, . . . , 1) 



GENERATION MODULO THE ACTION OF A PERMUTATION GROUP 



3 



Proposition 2.3. For any permutation group G C & n , for any integer vector v, 
if v is not canonical under G, all children of v are not canonical. Therefore, the 
canonicals form a "prefix tree " in the tree of all integer vectors. 

Sketch of proof: When a father is not canonical, there exists a permutation 
such that the permuted vector is greater. Applying the same permutation on the 
children shows also it cannot be canonical. 



(0,0,0) 



(0,1,0) (0,0,1) 

/ \ I 
{©A©} {%±r±} Whfy 

/ \ II 

(3,0,0) (2,1,0) (2,0,1) (1,2,0) (1,1,1) {©^ 




Figure 1 . Enumeration tree of integer vectors modulo the action 
of G = ((1, 2, 3)) C 63, the cyclic group of degree 3. 



Figure [T] displays integer vectors of length 3 whose sum is at most 3 and shows 
the tree relations between them. Choosing the cyclic group of order 3 and using 
the generation strategy, underlined integer vectors are tested but are recognized to 
be not canonical. Using Proposition [531 crossed-out integer vectors are not tested 
as they cannot be canonical as children of non canonical vectors. 

Our strategy consists now in making a breath first search over the sub-tree of 
canonicals. This is done lazily using Python iterators. 

2.2. Testing whether an integer vector is canonical. As we have seen, the 
fundamental operation for orderly generation is to test whether an integer vector is 
canonical; it is thus vital to optimize this operation. To this end, we use the work 
horse of computational group theory for permutation groups: stabilizer chains and 
strong generating sets. 

Following the needs required by applications, we want to test massively if vectors 
are canonical or not. For this reason, we will use a strong generating system of 
the group G. We can compute this last item in almost linear time |Ser03| using 
GAP |GAP97j . 

Let n a positive integer and G a permutation group G C & n . Recall that its 
stabilizer chain is G n — {e} C G n _i C • • • C G\ C Go = G, where 

V«, 1 < i < n : Gi := {g 6 G|Vj s$ i : g(j) = j} . 

From this chain, we build a strong generating system T = {T±, T2, . . . , T n } where 
Tj is a transversal of Gf-i/Gf. This set of strong generators is particularly adapted 
to the partial lexicographic order as stabilizers are defined with positions 1, 2, . . . , n 
from left to right. 

Let n and i be two positive integers such that 1 i ^ n. For v = (vi, . . . , v n ) 
and w = (wi, ■ ■ ■ , w n ) two integer vectors of length n, let us define the following 
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binary relations 

v <i w <^=^ (vi,...,Vi) <i ex (wi,...,Wi) 

v s^j w <s=^> (vi,...,Vi) < iex (wi,...,Wi) 

v =j w •<=>• Vj, 1 ^ j ^ i : Vj = Wj 

where <i ex and ^i ex represent regular strict and large lexicographic comparison. 

Algorithm[T]is a natural extension of McKay's canonical graph labeling algorithm 
as it is explained in (HR09 . 



Algorithm 1 Testing whether an integer vector is canonical 
Arguments 

• v: An integer vector of length n; 

• sgs(G): A strong generating set for G, as a list {Ti, . . . , T n } of transversals. 

def is_canonical(y , sgs(G)) : 
todo «— {v} 
for i £ {1,2, ...,n} : 
new_todo { } 
for w g iodo : 

children <S— {g ■ wjg G Ti} 
for c/iiid 6 children : 
if v <i child : 

return False 
else : 

if v =i child and child ^ newjtodo : 
newJbodo <— newJtodo U {c/iiZd} 
todo <— new-todo 
return True 



Algorithm [T] takes advantage of partial lexicographic orders and the strong gen- 
erating system of the group G. It tries to explore only a small part of the orbit of 
the vector v; the worst case complexity of this step is bounded by the size of the 
orbit, and not by \G\. In this sense, it does take into account the automorphism 
group of the vector v. 

Proposition 2.4. Let n be a positive integer and G a subgroup of & n . Let v be 
an integer vector of length n. Algorithm^ returns True if 'v is canonical under the 
action of G and returns False otherwise. 

Sketch of proof: Lt is based on the properties of a strong generating system. 

3. Complexity 

3.1. Theoretical complexity. 

3.1.1. Efficiency of the tree structure. Let n be a positive integer and G C & n a 
permutation group. For any non negative integer d, let C(d) (resp. C(d)) be the 
number of canonical (resp. non canonical) integer vectors of degree d. Based on the 
tree structure presented in Section liOl let T(n) (resp. T(n)) the number of tested 
(resp. non tested) integer vectors. 
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Proposition 3.1. Generating all canonical integer vectors up to degree d using 
the generation strategy presented in Section presents an absolute error bounded 
by C(d). Equivalently, regarding the series, we have 

d d 
i=0 i=0 

Sketch of proof: Using Lemma \2.3\. we get this bound noticing two tested but 
non canonical vectors cannot have a paternity relation. 

This absolute error is not very explicit (directly usable), but it can be used to 
get a relative error at the price of a rough approximation. 

Corollary 3.2. Let n and b be two positive integers and G C & n a permutation 
group. Generating all canonical monomials under the action of G up to degree d 
using the generation strategy presented in Section\^presents a relative error bounded 

. • rri(|G|-l) n 

bymm{ ^ ' ,n-l}. 

Sketch of proof: We use the previous proposition with the fact that any integer 
vector has at least one child but no more than n — 1 children (the generation root 
is the only one having n children). 

The bound is optimal for trivial groups ({e} C S n ), and seems to be better as 
the permutation group is of small cardinality. This relative error becomes better as 
we go up along the degree and tends to become optimal when the degree goes to 
infinity. 

3.1.2. Complexity of testing if a vector is canonical. We now investigate the com- 
plexity of Algorithm Q] We need first to select a reasonable statistic to collect, 
which will define the complexity of this algorithm. 

The explosion appearing in the algorithm is conditioned by the size of the set 
new_todo. For v an integer vector and {Ti, . . . , T„} a strong generating system of a 
permutation G, when i runs over {1,2,..., n} in the main loop, the set new-todoi 
contains at step i: 

new_todoi = {g 1 ---g i - \\g-y ■ ■ ■ ^ ■ v =j v, Vj < i : g 3 S Tj} 

The right statistic to record is the size of the union of the newJtodoi for all i 
such that the algorithm is still running: that corresponds to the part of the orbit 
explored by the algorithm. This statistic appears to be very difficult to evaluate by 
a theoretical way. However, collecting it with a computer is a simple task. 

3.1.3. Parallelization and memory complexity. Let us note that this generation en- 
gine is trivially amenable for parallelism: one can devote the study of each branch 
to a different processor. Our implementation uses a little framework SearchFor- 
est, co-developed by the author, for exploration trees and map-reduce operations 
on them. To get a parallel implementation, it is sufficient to use the drop-in paral- 
lel replacement for SearchForest under development by Jean-Baptiste Priez and 
Florent Hivert. 

The memory complexity of the generation engine is reasonable, bounded by the 
size of the answer. Indeed, we keep in the cache only the Canonical vectors of 
degree d — 1 when we search for those in degree d. In case one wants to only iterate 
through the elements of a given degree d, then this can be achieved with memory 
complexity O(nd). 
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3.2. Benchmarks design. To benchmark our implementation, we chose the fol- 
lowing problem as test-case. 

Problem 3.3. Let n be a positive integer and G C ©„ a permutation group. Iterate 
through all the canonical integer vectors v under the staircase (i.e. Vi < n — i). 

A vector v of length n is said to be under the staircase when it satisfies v ^i ex 
(n- l,ra-2,. ..1,0). 

This problem contains essentially all difficulties that can appear. The family of 
n ! integer vectors under the staircase contains vectors with trivial automorphism 
group as well as vectors with a lot of symmetries. Applications also require to deal 
with this problem as the corresponding family of monomials plays a crucial role in 
algebra. 

3.2.1. Benchmarks for transitive permutation groups. We now need a good family 
of permutation groups, representative of the practical use cases. We chose to use the 
database of all transitive groups of degree < 30 I lull)-")! available in Sage through 
the system GAP [GAP97] . 

The benchmarks have been run on an off-the-shelf 2 . 40 GHz dual core Mac Book 
laptop running Ubuntu 12.4 and Sage version 5.3. 

3.3. Benchmarks. 

3.3.1. Tree Structure over integer vectors. This first benchmark investigates the 
efficiency of the tree structure presented in Section |2~T1 As we don't test children of 
non canonical integer vectors, one wants to take measures of the part of tested non 
canonical vectors (which corresponds to the useless part of computations). For that, 
we solve Problem 13.31 for each group of the database and we collect the following 
information as follows. 



Transitive Groups of degree 5 


Database Id. 


\G\ 


Index in Sn 


Canonicals 


number of tests 


1 


5 


24 


71 


81 


2 


10 


12 


68 


81 


3 


20 


6 


46 


67 


4 


60 


2 


41 


67 


5 


120 


1 


41 


67 



This table displays the statistics for transitive groups of degree 5. Database Id. is 
the integer indexing the group, \G\ and Index in & n are respectively the cardinality 
and the index of the group G in the symmetric group & n . Canonicals denotes the 
number of canonical vectors under the staircase and number of tests is the number 
of times the algorithm testing if an integer vector is canonical is called. 

From this information, we set a quantity Err defined as follows: 

number of tests — Canonicals 
Canonicals 

The following figure shows Err depending on the index ^ . The figure contains 
166 crosses, one for each transitive group over at most 10 variables. We use a 
logarithmic scale on the x axis. 
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Figure 2. Relative Error between number of tested vectors and 
number of canonicals vectors. 



3.3.2. Empirical complexity of testing if a vector is canonical. Algorithm [T] needs to 
explore a part of the orbit of the tested integer vectors. The following table displays 
for each transitive group over 5 variables, the number of elements of all orbits of 
tested vectors solving Problem 13 . 31 compared to the total number of integer vectors 
explored. 



Transitive Groups of degree 5 


Database Id. 


\G\ 


Index in @ n 


total orbits 


total explored 


1 


5 


24 


401 


351 


2 


10 


12 


691 


393 


3 


20 


6 


1091 


365 


4 


60 


2 


1891 


328 


5 


120 


1 


1891 


326 



Now we define Ratio to be the average size of the orbit needed to be explored 
to know if an integer vector is canonical: 

total explored 

Ratio := . 

total orbits 

The following figure plots Ratio in terms of \G\ for transitive groups on at most 9 
variables. 
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Figure 3. Average, over all integer vectors v under the stair case, 
of the number of vectors in the orbit of v explored by 
is_canonical (v) . 



3.3.3. Overall empirical complexity of the generation engine. We now evaluate the 
overall complexity by comparing the ratio between the computations and the size 
of the output. We define the measure Complexity as follows: 

. total explored 

Complexity := — — : — - — . 

Canonicals 

The following graph displays Complexity in terms of the size of the group \G\ for 
transitive Groups on up to 9 variables (and excluding the alternate and symmetric 
group of degree 9). 
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The dashed line has as equation y = 5Zn(|G|). Therefore, we get the following 
empirical overall complexity: 



Computations = 0(ln(\G\) x Output size) 
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3.3.4. Tests around the unlabeled graph generation problem. Although the genera- 
tion engine is not optimized for the unlabeled graph generation problem, we can 
apply our strategy on it. 

Fix n, and consider the set E of pairs of elements of n. The symmetric group 
& n acts on pairs by a ■ = (a(i), a(j)) for a £ & n and S E. Let G be the 
induced group of permutations of E. A labeled graph can be identified with the 
integer vector with parts in 0, 1. Then, two graphs are isomorphic if and only if the 
corresponding vectors are in the same G-orbit. 

Now, one needs just to know which are these permutation groups acting on 
pairs of integers. In the following example, we retrieve the number of graphs on n 
unlabeled nodes is, for small values of n is given by: 1, 1, 2, 4, 11, 34, 156, 1044, 
12346, 274668, 12005168, ... 

sage: L = [TransitiveGroupd , 1) , TransitiveGroup(3 , 2) , 
TransitiveGroup(6,6) , TransitiveGroup(10 , 12) , TransitiveGroup(15 , 28) , 
TransitiveGroup(21,38) , TransitiveGroup(28,502)] 

sage: [IntegerVectorsModPermutationGroup(G,max_part=l) . cardinality () for G 
in L] 

[2, 4, 11, 34, 156, 1044, 12346] 

Notice that our generation engine generalizes the graph generation problem 
in two directions. Removing the option max_part, one enumerates multigraphs 
(graphs with multiple edges between nodes). On the other hand, graphs corre- 
spond to special cases of permutation groups. From an algebraic point of view, we 
saw graphs as monomials whose exponents are or 1, canonical for the action of 
the symmetric group on pairs of nodes. 

4. Computing the invariants ring of a permutation group 

Let us explain how the generation engine from Section [5] is plugged into effective 
invariant theory (see [DK02] and |Kin07j ). 

A well-known application to build an invariant polynomial under the action of 
a permutation group G is the Reynolds operator R. From any polynomial P in n 
variables x := x\, x 2 , ■ ■ ■ , x n , the invariant is 

where a ■ P is the polynomial built from P for which a has permuted by position 
the tuple of variables (x\, x 2 , ■ ■ ■ , x n ). Formally, for any a G G 

((T • P)( Xl ,X2, ...,X n ) := P(x a -i {1) ,X a -i {2} , . . . , X a -i (n) ). 

For large groups, the Reynolds operator is not very convenient to build invariant 
polynomials. If P is a monomial x a := x^x^ 2 • ■ ■ X®" where a = (oi, 02, ... , a n ), 
the minimal invariant one can build in number of terms is the orbit sum (x a ) 

Orb(<£>) 

of X. 

Let K a field, we denote by K[x] the ring formed by all polynomials invariant 
under the action of G. 



K[xf := {P e K[x]|Vct e G : a ■ P = P}. 
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For any subgroups G of & n and K a field of characteristic 0, a result due to 
Hilbert and Noether state that the ring of invariant K[x] G is a free module of rank 
t^tt over the symmetric polynomials in the variable x. Computing the invariant 
ring K[x] G consists essentially in building algorithmically an explicit family (called 
secondary invariant polynomials) of generators of this free module. 

Searching the secondary invariant polynomials from orbit sum of monomials 
whose vector of exponents is canonical (instead of all monomials) produces a gain of 
complexity of \G\ if we assume that all orbits are of cardinality \G\ . This assumption 
is obviously false; however, in practice, it seems to hold in average and up to a 
constant factor [Borllj ). 

In [BTllj . the authors calculate the secondary invariants of the 61 s * transi- 
tive group over 14 variables whose cardinality is 50803200. Using the canonicals 
monomials, they managed to build a family of 28 irreducible secondary invariants 
deploying a set of 1716 secondary invariants. This computation is unreachable by 
Grobner basis techniques. 



5. Computing primitive invariants for a permutation group 

5.1. Introduction. We now apply our generation strategy to this problem con- 
cerning effective Galois theory 



Problem 5.1. Let n a positive integer and G a permutation group, subgroup of 
6 n . Let K be a field and x := x\, . . . , x n be n formal variables. Find a polynomial 
P G K[xi, . . . , x n ] such that 



{ae& n \a-P = P} = G. 



A such polynomial is called a primitive invariant for G. 

Problem 15.11 (exposed in Gir87] and [AbdOOj ) consists in finding an invariant 
P under the action of G such that its stabilizer Stabe n {P) in & n is equal to G 
itself. Solving this problem becomes difficult when we want to construct a primitive 
invariant of degree minimal or a primitive invariant with a minimal number of 
terms. 
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Algorithm 2 Primitive invariant using stabilizer refinement 
Prerequisites : 

• IntegerVectorsModPermgroup: module to enumerate orbit representatives; 

• stabilizer _of_orbit_of(G,v): a function returning the permutation group which 
stabilizes the orbit of v under the action of the permutation group G. 
Arguments: 

• G: A permutation group, subgroup of & n . 

def minimal_primitive_invariant(G) : 

cumulateStab <— SymmetricGroup(degree(G)) 
chain <s— [[(0, 0, . . . , 0), cumulateStab, cumulateStab]] 
if Cardinality {cumulateStab') —— Cardinality(G) : 

return chain 
for v 6 IntegerVectorsModPermgroup(G) : 

AutV «— stabilizer-of-Orbit_of(G, v) 

Intersect <— cumulateStab n AutV 

if Cardinality (Inter sect) < Cardinality (cumulateStab) : 
chain <— chain U [v, AutV, Intersect] 
cumulateStab 4- Intersect 

if Cardinality [cumulateStab) —— CardinalityiG) : 
return chain 



5.2. Primitive invariant of minimal degree. 

5.3. Benchmarks. Algorithm [2] terminates in less than an hour for any subgroup 
of 6 io- Even, it can calculate some primitive invariants for a lot of subgroups with 
degree between 10 and 20 while the literature only provides examples up to degree 
7 or 8. Using the same computer, this benchmark just collects the average time in 
seconds of execution of Algorithm [2] by executing systematically the algorithm on 
transitive groups of degree n. 



Degree of Groups 


1 


2 


3 


4 


5 


6 


7 


8 


9 


Computations time 


0.008 


0.064 


0.104 


0.160 


0.208 


0.393 


0.537 


2.364 


27.093 



We would like to thanks Nicolas M. Thiery, Simon A. King, Karl-Dieter Crisman 
and Dmitri V. Pasechnik for useful comments about implementation details, review 
of code and Cython optimizations. 

This research was driven by computer exploration using the open-source mathe- 
matical software Sage [S + 09j . In particular, we perused its algebraic combinatorics 
features developed by the Sage-Combinat community SC c08j . as well as its group 
theoretical features provided by GAP [GAP97 . 
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